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ABSIRICI 

This paper reports the results of a case study of the 
process of formative evaluation in the development of one unit of 
Individualized Science (IS). It is an analytic description of the 
curriculum pro ject' s experiences in formative evaluation rather than 
an evaluative critique. Following an introductory explanation^ the IS 
program and the IS program directors* conception of formative 
evaluation are discussed* The concluding sections analyze the 
activities observed (both in the specific case and in general terms) , 
the manifest and latent functions of formative evaluation, and some 
structural implications of these functions* (Author) 
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Abstract 



This paper reports the results of a case study of the procesi of forniatlve 
evaluation in the development of one unit of Individualized Science (IS), It 
is an analytic descfiption of the currieulum project's experiences in forma- 
tive svaluation rattier kian an evaluative critique^ Following an introduce 
tpry explanatory sectionp the IS pFogram and tht^JS program directors' 
conception of formative evaluation are discussed^ The concluding sections 
analyze the activities obs^^rved- -both in the specific case and in general 
terms, the manifest and latent functions of form.ative evaluation, and soma 
structural impUeatlons of these fu^ctions'p Th^ intended audience are fliose 
engaged in curriculum development, in formative evaluation, and o^er 
students of knowledge production and utilization. 
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FORMATIVE EVALUATION AS A SOCIAL PROCESS^ 
A CASE STUDY 



I^eelie Salmon- CoK and Burkart Holzner 

Learning Research and Development Center 
University of Pittsburgh 



The research reported herein comes irom a eaie study of formative 
evamation, a domain of activity characteristic of educational design and 
development. Formative evaluation Is a mechanism to assess the adequacy 
of instructional materials and procedures as they are taking form. It is a 
technical tool for producing better curricula. In this study, however, we 
were not concerned with the substantive outcomes of the application of the 
process of formative evaluation. We were not concerned with whether or 
not the materials being evaluated were judged -good" or ''bad" or something 
in between. O^ir focus was on the social processes of formative evaluation 
as they evolved in a structurally defined setting* 

The authors of this report are sociologists whose theoretical frame- 
works are rooted in the sociology of knowledge. It is the intention of such 
sociology to understand both the mechanics of knowledge production and the 
meanings attached to those mechanics in the minds of the actors involved. 
In pursuing these objectives, our methods are similar to those of the anthro- 
pologist conducting field work* 

There were three distinct purposes guiding this research study. First 
was the need for a fine-grained. description of the formative evaluation proc-r 
ess. What actually occurred during that time when the curriculum designers 
were engaged in development and prototype testing and simultaneously con^ 
ducting evaluation they labeled "formative? Second, given the activity 
description we constructed, what kind of comparison could be drawn be- 
tween this and the developers' description of their own activity? Our third 
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purpose was the analysis of what had been observed to articulate the basic 
components of the proce-aee involved and their interreUdonship, This 
analyiiE dealt with three aspects^ (a) problem^deteetion procedurea, 
(b) decision-making procedures, and (c) the manifest and latent functions 
of the evaluation process. 

Before discussing the details of the study, it is useful to deicHbo 
briefly the organization in which the research took place and the role the 
authors play In the orgBnlzation. The research was undertaken at the 
Learning Research and Developrnent Center {UT.DC) at the University of 
Pittsburgh. LRDC isl^institute designed to bring behavioral science to 
bear on education for the purpose of Improving practice in the schools. In 
particular. LRDC is concerned %vith designing environments for learning 
that are adaptive to the individual characteristics of students. As part of 
this work, the Center has developed a number of new curriculum programs 
for the preschool and elementary school. 

As Booiologists. the authors' major role within the Learning Research 
and Development Center is to study the organization as a whole and to report 
to aft external Board of Visitors who monitor and guide the Center in its . 
activities. In the conduct of studying events and trends in the Center. 
Salmon-Cox was attending a series of meetings being held to plan the multl- 
facted evaluation of Individualized Science (IS), an elementary school sci- 
ence program under development at LRDC. This innovative evaluation 
strategy, which encompassed studies done from several perspectives and 
resulted in an array of various kinds of informatloa, is described by 
Leinhardt (1975), who planned and coordinated the effort. In November 
1974. the IS project directors proposed that, as part of the larger evalua- 
tlon effort, the authors conduct an intensive study of the process of forma-, 
tive evaluation a. It is carried out by the IS project. It was p-oposed that 
we study the prototype teat and formative evaluation of the "Quetclet" unit 
(a unit on statistical concepts) of the IS curriculum which was planned for 
Spring 1975. 
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Our reseE^r^h was conducted in the Spring and eariy Summer of 1975* 
Ab sociologists, we employed the methodology of obieivation, interviewj 
document analysis, and attendance at all possible meetings, as well as 
conducted some in-classroom observation to collect data for the study. 
Hiough the formative evaluation itself had not been completed* sufficient 
data were collected to provide material for the present analysis- 

In order to provide a context for our analysis of the processes of 
formative ev^aluation* we begin this paper with a description of the Indi- 
vidualized Science program and the procedures of formative evaluation as 
outlined by the IS project directors (ChampagnG ^ Klopfer, 1974)* In addi- 
tion to defining our domain of inquiry » their view of formative evaluation 
provides a mosaic to which later descriptions of actual activity can be 
compared. 

In the next sections of the paper we present the data base for this 
research- - the development of formative evaluaLion processes by the IS 
project- We analyze the activities undertaken* focusing on the social 
organization that performed the evaluative taskp and discuss the manifest 
and latent functions of the particular formative evaluation under study. In 
the concluding section, we reflect upon the total process observed, on the 
functions of formative evaluation In general, and on some structural impli- 
cations of those functions. We turn now to an overview of the Individualized 
Science program, 

"Q^g^PjQM jJnder Evaluations Individual i zed Science 

Audrey Champagne and Leopold Klopfer are co-direcEorB of the Indi- 
vidualized Science project and co-developers of IS. In their paper entitled 
"Individualized Learning in Individualized Science '' (1975), they de^scribe 
the program as followgi 

Individualized Science (IS) [is] an elementary^school science 
program . * * designed to serve children in present school 
grades K through 8. 
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lliere are Bi'vpn levels of the IS program, eaeh oi which con- 
sists of approximately one year's work in gcience* The sci- 
ence content of IS is drawn from both the biological and the 
physical sciences. One unique aspect of the prog ram is its 
emphasis on the cultural aspects of science and the inter- 
actions of science and society* IS is a comprehensively 
individualised elementary- school science program. Not 
only does IS provide mechanisms for a child to help plan 
his or her science activitieSi to manage his or her own in- 
structional materials, and to take part in the assessment of 
his or her learning, but it also provides opportunities for 
the child to make selections from among alternative learn- 
ing resources and from among alternative units of study* 

(p- n 

The authors specify five goals for the IS program* Tiliese goals are,' 

1* Scientific literacy goal ; The student acquires a founda- 
tion for scientific literacy, 

2, Student jelf -direction go al; The student views the learn- 
ing process as primarily self -dir ected and self-initiated* 

3, Student co-evaluation goal ^ Thm student plays a major 
role in evaluating the quality, extent and rapidity of his 
learning- 

4, Affective goal r The student displays informed attitudes 
toward his study of science, scientific inquiry, and the 
scientific enterprise, 

5, Inquiry goa l; Hie student becomes skillful in using the 
processes of scientific inquiry and is able to carry out 
inquiries, (pp^ 7-8) 

In discussing these goal.^, they point out that three of the program-s 
goals are related to learning science content and two are related to the 
student*s management and evaluation of his or her own science learning* 
■'Each of the five goals Is subdivided into level competencies and the sci- 
ence subject matter learnings ave specified in unit behavioral objectives* 
TJie analysis of goals into competencies and behavioral objectives has 
guided the development of the learning resources of IS" (p* 8), The co- 
developers go on to describe 19 different kinds of learning resources in 
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IS which diffBr in "function, coniponentSf insfcructional strategy, primary 
senee modality, and the social setting in which they mre used. " ' . 

In the Quetelet unit on gtatistical concepts, the students collect and 
record data and perform iome statlitical analyses. During the prototype 
testing phase of this unit, a number of learning resources were being 
evaluated; Readings in Science (RIS), illustrated booklets containing sev- 
eral readings^ optional read-along tapes, wifli eomprehenslon and exten- 
sion questions accompanying the readings; Invitations to Explore (ITEs), 
illustrated booklets with printed texts, optional read-along tapes, manipu- 
lative materials, and a response sheet booklet: Student Activities (SA)^ 
printed cards which describp activities, practice sheets, and manipulative 
materials. These resources in the Quetelet unit are reforred to in this 
paper in abbreviated form. 

Formative Evaluation in Curriculum Development i 
The Rationale 

An article entitled "Formative Evaluation in Science Curriculum De- 
velopment'' written by Champagne and Klopfcr (I9'^4) sets forth their philoso- 
phy and outlines procedures for formative evaluation- TTiey stress that 
evaluation is an integrai part of the development process. Their philosophy 
stems from the distinction made by Cronbach (1963) and Scriven (1967) be- 
tween "formative" and "sumtrsative" evaluation. Formative evaluation is 
carried out while the program is being developed and the findings of the 
evaluation serve as a basis for revision of the program. Summative evalua- 
tion takes place after the program "goes out to the public" and changes 
based on the findings of this evaluation cannot be made qasily* 

Champagne and Klopf^^f state that formative evaluation must deter- 
mine the effectiveness of the program=-the outcomes which are explicitly 
planned. But questions of effectiveness, the focus of most evaluation 
studies, "are far from being the only ones that must be considered. " They 
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believe forrnative evaluation mugt also determine the effects of the program- 
the outcomes which are not among specified aims. 

The authors propose four stages of formative evaluation which span a 
continuum from the initial conceptual phase through the final implementa- 
tion and teacher preparation phase. During Stage A, IS staff niembers focus 
on conceptualizatioa and planning and it is during this stage feat the instruc- 
tional materials are designed and reviewed before being tried out in a class- 
room. During Stage B, the prototype testing stage, the materials are tried 
out in one of LRDC's associated developmental schools. The Stage B evalua- 
tion is conducted by school personnel and members of the IS development 
group and revisions are made based on the information they obtain. In 
Stage C, the materials are implemented and evaluated in a network of field 
test schools. Stage D evaluation centers on tiie effectiveness of the teacher 
preparation program, fee ease with which the program is implemQntQd, 
and marketability. Data from Stages C and D are not collected by IS or 
LRDC staff but by Research for Better Schools (RBS), an educational labora- 
tory located in Philadelphia which has been responsible for field testing and 
dissemination of several programs developed at LRDC, It should be noted 
that Stages C and D are different in character from A and B which Involve 
only LRDC staff and a classroom in an associated school, 

Ihe authors have developed a aeries of questions to be addressed at 
the varying stages, Ihe questions are organized into six categories, each 
of %.nich focuses on a different aspect of fee formative evaluation: (a) con- 
ceptualisation and planning, (b) student instructional materials, (c) student 
behaviors with respect to the science content, (d) student behaviors in the 
classroom environment, (e) the teacher, and (f) marketability. The authors 
admit it may be impossible to address the multitude of questions they have 
posed (as many as 40 in one category) in any single formative evaluation. 
But they suggest that an adequate samplinq of each of the questions in each 
category is essential, lliey point to three criteria for question selection.- 
One should probe the most innovative aspects of the program, the aspects 
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of the program which its developers think most iignificant and Interesting, 
and thOie areas which provide relevant data, the collection of which is not 
too difficult* 

Perusal of the multitude of questions shows that they do Indeed span 
a wide range of program effects. In addition to the more usual questions 
regarding content selection and instructional strategies, there is also a 
great deal of emphaeis on inquiry skills, seLf ^management and self ^direction, 
and the overall culture of the classroom as it is created by and contributes 
to the program's functioning. Tlie students' and teachers' feelings about 
the materials are considered along wiUi the more pragmatic concerns of 
packaging, efficient storage, costs, and such. 

In their article. Champagne and Klopfer appear to posit role dif- 
ferentiation between designer and evaluator^ that is, they do not seem to 
expect that these two functions will be conducted by the same person or 
people. Potential problems resulting from this role differentiation are 
anticlpated-^problems of responsibility and relative authority. They dis^ 
cuss, from their point of view as designers, the kind of relationship that 
they feel must exist between an evaluator and a curriculum designer. 
They state that an evaluator should present data "in a thoughtful way, a 
way that is honest and yet does not threaten or antagonize the developer, " 
The authors consider the findings from formative evaluation confidential, 
n_m formative evaluation "should remain privileged Information with ^le 
curriculum development organization, where it can be meaningfully inter- 
preted. " They note that the "authority to make mDdiflcations in the pro- 
gram and to decide on the nature of the modifications resides with the 
developers" (1974. p, 202), 

It is interesting at this point to note that the developers of IS par- 
tially solved the role differentiation problem by designing and conducting 
their own formative evaluation at Stages A and B (as they have defined 
these). At later stages, they keep a close watch over the evaluative 
activities of others. 
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The Developnient/Evalu ation Process 

The development of the Quetelet unit, from conceptualisation until 
prototype testing, eovered a perioy of about two yearsp Our study of the 
IS project began late in that tinie, but our researeh allowed us to recon- 
struct events and the involvement of salient aetors from the unites incep- 
tion* Such a reconstruction was essential for our analysis, plaoing what 
we could observe Into the larger context of what had occurred from the 
start. 

Hie Quetelet unit is a mainstream^ -i. e* , mandatory, basic core- = 
unit of Level G of I S. TTiis implies that it will be used mainly by seventh = 
and iighth^grade students; however, since IS is an individuali^ded program 
students at other grade levels can work in the unit, line bulk of the cur- 
riculum materials for the unit were written in 1973, In 1974^ wi^_ editing 
and rewriting, the instructional materials wure completed. In April 197^, 
when the prototype version was produced by the publisher, implementation 
of the unit began in an eighth-grade classroom at St. Mary's school, a 
Center -as sociated school located in Pittsburgh. 

Expectations Concerning Evaluation of Q uetelet 

TTie general consensus among those who saw Quetelet before its use 
in the school was that this was one of the best prepared, most polished 
units ever to be entering prototype testing in IS. Tliere was, however, 
one anticipated source of problems with the content* This was tiie ques- 
tion of the level of mathematical sophistication needed to work in Ouetelet. 
Designed to teach basic statistical concepts to seventh and eighth graders, 
the unit appeared to require an abillLy and ease with numerical manipula- 
tion which some doubted the students would possess. Klopfer dismissed 
this possibility, »saying he thought there was a good match between the 
students' madiematical level and what was necessary for Quetelet. 
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What was found on this question ii an interesting example of the con- 
straints placed by the "real world" on curriculum design, testing* and 
evaluation. The anticipated problem during the prototype teat' at St, Mary's 
could not be inveitigated because the bulk of St. Mary^s eighth graders who 
were working on Quetelet had a strong background in the major concepts of 
the unit* Informal discussion among the school and IS project staff involved 
seemed to indicate that everyone suspected diat this kind of student back- 
ground was out of the ordinary* Eighth-grade mathematics classes do not 
normally cover statistical concepts. However, of ^e 18 students to work 
on Quetelet in the Spring of 1975, 13 were in a "high maai track, and 
found much of the substantive content of Quetelet to be a repeat of work 
done in their mathematics class in the same academic year. The remain- 
ing five students were in an average mattiematics class, and they, too, 
seemed familiar with many of the concepts, 

A course of action was outlined to remedy this problem. They 
planned to collect further data in the fall on a new group of *'low mathe- 
matics" eighth graders and on seventh graders, with the hope, or ejcpecta- 
tion, that these groups would not be prefamiliar with the content. Also, 
the people at RBS responsible for the field testing of Quetelet would be 
alerted to the possible occurrence of difficulty with the mathematical 
level, as well as to the apparent inability of the Pittsburgh prototype test 
to investigate this question, 

Ariother more general anticipation wag that because Quetelet was in 
"prototype" form, it would pose certain problems. This worry was ex- 
pressed by the assistant teacher, Karen Evans, who had not been involved 
in prototype testing before. Since the unit had never been in the classroom 
before, she feared it might be in very rough forni. In particular, she was 
concerned that th& necessary supplies would not arrive on time, the printed 
materials would not be clear, and something might go wrong with the manipu- 
iables. Sister Martha, the teacher at St. Mary's had been involved in 
prototype testing of another unit of the IS program and had experienced a 
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; number of problems. It would not, therefore, have been purprlsing if she 
harbored some fears about working through Quetelet also. If she did, 
however, she did not express them to the observer* 

By mid-May, after about a month of implementation, Evans reported 
to the observer that the problems expected were not appearing. Not only 
were there none of the anticipated problems with the unit, but no unantici- 
pated trouble spots had arisen either^ The prototype materials of Quetelet 
were in order; in fact, ihe found that she had more problems with "already 
done" units of IS. Because of the students* background in math, her expec^ 
:tatlon that she would be required to provide a great deal of explanation was 
not being realized* Things were going so smoo^ly, she said, that it was 
"almost boring* " The children were highly motivated. They were eager 
to work in the unit to help out the people from the University, and they 
also seemed to enjoy the content* 

Anticipatlc3ns of the Formative Evaluation Session 

In April, the IS staff began collecting observational data on students 
working in Quetelet. The paper by Champagne and Klopfer (1974) on forma- 
tive evaluation was reviewed, and several groups of questions which would 
guide observation were selected. The questions were selected with refer- 
ence to three criteriai (a) the answers to the questions were obtainable 
through observation; (b) the questions were relevant to observation of a 
single unit, rattier than the who^e program; and (c) they were appropriate 
to this phase of student use, namely prototype testing. Data collection 
proceeded through April, May, and part of June. 

On July 7, 1976, a meeting was held at LRDC to discuss the Quetelet 
unit, bringing together the directors of IS, the IS staff responsible for coU 
leetlng observational data during the evaluation, the classroom teacher, 
the assistant teacher, and a student from St* Mary's eighth grade who had 
completed the Quetelet unit. This meeting was a focal point of the forma- 
tive evaluation of the unit. ^ ^ 
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People who had been involved in previoug formative evaluations were 
asked how the long review meeting at LRDC wa^ conducted and what uaually 
Qccurred, There had not been many guch in the past year or so* Thm only 
fairly recent one had been of the IS *'Harvey'' unit, where materials had 
undergone some significant changes^ 

In the Spring, Sister Martha^s comments about the upcoming session 
on Quetelet were very generaL She was vague about what details she might 
be questioned on, but did say that in the past she had critiqued the materials 
for consistency in instructions given to the students and clarity. She phrased 
her anticipated contributions primarily In affective terms, putting little em- 
phasis on any role she might play in content evaluation. She said that the 
LRDC staff always seemed highly receptive and grateful for whatever com- 
ments she made. However, she did convey a picture of a session in which 
she and Karen Evans would play major roles, reporting on their classroom 
experience with Quetelet. Evans, having only one yearns experience with 
IS, had no specific eKpe.ctations concerning the July 7th meeting. 

L.RDC staff were also fairly imprecise in their expectations. They 
also gave the impression that Sister Martha and Karen Evans would have the 

major roles in the meeting, ITierefore, what generally seemed to be ex- 

pected by the participants was an informal meeting consisting essentially of 
questions asked by LRDC staff and answered by school personnel. Supple- 
mental information would be provided by IS project staff observers when 
necessary. 

Formiatiye Evaluation Ses^sion 

The July 7th meeting was held in the midafternoon and lasted for about 
two and one-half hours. Contrary to expectations, it was fairly formal, 
structured, and run by an LRDC staff member, Joan Fogarty, Fog arty was 
involved in some of the first writing for Quetelet and was in charge of the 
unit's formative evaluadon. Inputs from the two teachers were minimal; 
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from IS staff aislsting Fogarty in coUecting observational data* there were 
none at alL 

A number ot factors set the tone for the meeting and probably made 
the diHCuseion goinewhat atypicaL First, insufficient data had be^n col- 
lected before school closed, so this session was not really a "wrap-up* " a 
final formative evaluation occaslrTn, but rather an interim meetings Secondj 
it was clear to all concerned before the meeting was held that Quetelet was a 
fairly unproblematic unit* It was well written, had been working smoothly 
in the classroom, and seemed to require only minimal revisionB. In addi- 
tion, the presence of an eighth-grade student probably also affected the tone^ 
Apparently^ student participation had been talked of before, but this was the 
first time it actually occurred, 

Joan Fogarty opened the meeting and spent the first hour discussing 
the preliminary nature of the evaluation and the questions that had guided 
observation^ She said that the students had been observed and talked with 
in an informal^ participatory way. She shared with all present the antici- 
pated problenis of the unit's mathematical level, and the fact that this could 
not be evaluated because of the familiarity most of the students had in the 
mathematical concepts of the unit* She also pointed out that a number of 
instructional materials- -many of the RISs and S As and some of the ITEs-- 
had not been used by any students and some by only one or two* lliis fact, 
coupled with the small number of students and the homogeneity of the stu- 
dent sample, made the data Incomplete* Since the mathematical level of 
the unit had been the major anticipated source of difficulty, and since data 
on it were inconclusive, it served as a background for the discussion which 
followed, being referred to in several conteKts. 

For the remainder of the meeting, conversation ranged over details 
of the unit, Tliere was discussion of specific ITEs, and the meeting ended 
with some general talk of "where do we go neKt* " iTie specific details dis- 
cussed uncovered several points for revision, none of them implying radical 
: change. Table 1 lists the specific problems raised at the meeting , indicating 
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how they were diacovered, and tho recommendations put forth by tht- evalua- 
tion group for their remedy* A brief review of three of the prpblems raised 
will give a feeling for how they emerged in the course of the meeting antd 
how the respective sQlution related to the goals mnd philosophy of IS. 

The first problem listed in Table l--the nonuse of some manipulables- 
was first raised by Fogarty' Sister Martha and the student agreed that the 
prohleni had occurred, Thm reason given for their nonuse was that the stu- 
dents either completely understood or could mentally visualise the ooncepts 
being taught without propi. Champagne suggeeted ^'dropping these manlpula- 
bles^ Discussion followed, Klopfer summed up the discussion and the 
result was a decision to leave them in the unit because other^ less-prepared 
students might find them useful* 

The decision to leave the manipulables in the program was an inter- 
esting onej the reason for which can be found in the underlying philosophy 
of the developers* The developers view IS as a multifaceted program that 
matches instruction to the various characteristics of individual learners 
(Champagne ^ Klopfer, 1975)* llius, IS has been designed so thAt each 
unit has a wide variety of learning resources (e* g* * illustrated lesson book- 
lets, audiotapeSj exploration kits which contain manipulative materials and 
simple laboratory equipment, etc* ) to try to assure maKimum accommoda- 
tion to individual differences* In the face of disconfirming evidence of the 
utility of some of the manipulatives- -not a single student had used them- - 
^e developer's notion of their place in the instruction was unshaken* 

Hie fourth problem- -the lack of enough specified student-teacher 
conferences- -was raised by the student at the meeting in response to ques- 
tioning about what she would want to see done differently in the Quetelet 
unit* The problem was discussed at length as it had been a source of diffi- 
culty before with respect to earlier units, The publisher of the IS program 
had asked that more conferences be added to earlier units because as 
Champagne pointed out, the publisher felt that everything should be highly 
specified* Champafne said that in IS, J'we want to give kids and teachers 
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suggestloni* " The number of student- teacher qonferences Is ''the kind of 
decision a teacher makes, " The resolution of this problem was twofoldi 
Specify soma additisnal optional student- teacher cQtiferences, and strongly 
encourage atudents to seek help whenever they need it* 

A final interesting point stemming from the discussion of problems 
had to do with the "relevance" of Quetelet* the sixth question in Table 1* 
The seventh and eighth graders' inability to perceive much relevance in 
statistical concepts is not totally surpriBing when we consider college stu- 
dent plaints on the same subject* However^ the materials in Quettelet are 
specifically designed to make "real world" applications more obvious, Thm 
developers' decision was to return to ^is question when mora students had 
worked tiieiif way through the entire unit. 

In the closing minutes of the meeting, Sister Martha and Karen Evans 
bf^th spoke of the aspects of Quetelet that they particularly liked. Sister 
Martha pointed out diat Quetelet' s activities gave students a chance to "find 
out what-s behind what comes out of a science laboratory* " Evans said that 
die emphasis on data recording and data collection helped to tie together 
much of the preceding instructional material of IS* There was general 
discussion of the great intellectual sophistication of contemporary seventh 
and eighUi graders, making more salient die need for individualized pro- 
grams of high quality such as IS. The meeting closed wiUi consensus on 
the need to examine further the specific problems raised regarding particu- 
lar Instructional materials and teaching strategies and the student-teacher 
conference question* 

Analysis of the Process of Forniative Evaluaiion 

A major concern expressed by Champagne and Klopfer (1974) in their 
paper on formative evaluation was the role distinction between designer and 
evaluator* In the formative evaluation of Quetelet, such a distinction did 
not exist for the IS project during f^e stage of prototype testing. Many of - 
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the IS itaff were lii*?olved in this evaluation* There wai no member of the 
staff whose sole reiponeibility was the evaluative activity. 

. The model or structure of procedures which they preseut in their 

; paper waa applied by Klopfer and Charnpago»4 in the ^onduct of the actual 
formative evaluation of Quetelet, modified by ihc cognitive atylee of the 
actors fchemse'lves. Although they preaented groups of questions and stages 
i of activity as a model* it is apparent that, in fact^ a far more holistic 
apprbach was taken flian is conveyed in the paper^ By holistic wq mean an 
approach that places greatest emphasis on total configurations, on how 
L components fit widi one another to make a coherent whole. This might be 
contrasted with a model of evaluation or of activity which assesses the pre- 
cision with which any individual piece fuifilla its particular objective and 
assumes that a collection of such individual "eKcellences" makes for an 
excellent whole* 

This notion of a holistic approach is a consistent theme which perme- 
ates aH activities connected with I S- -curriculum design, internal project 
organisation, project/other agency interaction, and evaluation of materials* 
There is more of an emphasis in IS materials on process than on specific 
content. Anyone who has looked at the IS materials cannot but be impressed 
with the quantity and complexity of science content in this elementary school 
program^t -'scientific literacy'' is the first of the program's five goals* How- 
ever^ the other four goals emphasize style of interaction among students, 
teachers, materials, and the learning environment rather than mastery of 
science facti* 

The IS program is considerably more than a sequence of specific 
learnings in science. It Is a model for interaction and for a cultural en- 
vironment which fosters self -initiative and emphasizes the aesthetic aspects 
. ©f science* This empKasis on the total configuration of the program was 
evident in the evaluation process as well, where concern was more 'for con- 
gruency of the unit with the classroom and with the entirety of IS than for 
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specific details of content learning* Hiis is not to say that gpeciflc content 
is overlooked. It ig carefully icrutlriised, but as a means to a more com- 
plex end. 

Because there was so little ''wrong" with Quetelet, it is difficult to 
give examplei of this approach using that unit^ However, a previous experi 
ence with formative evaluation on the IS Harvey unit (a biological unit on 
circulatory systems) is illustrative. As part of Harvey^s resources, there 
was a series of Mln-Exs- -activities which required that the studeiits build 
physiological models of certain biological systems. During formative evalu 
tion, it was decided that these Mln-Exs would have to be replaced, Klopfer 
reported that the activities were "dull, " an assessment agreed to by the 
teachers and the LRDC staff. He said that on examination these Mln-Exs 
were, in and of thems elves, perfectly all right but that the children did not 
get: excited over them* More importantly, they did not support the overall 
philosophy of IS (especially the emphasis on inquiry and self -direction) as 
well as they might have. New ones were developed. 

Examination of the old and new Min-Exs reveals that though both are 
model-building activities, the new ones stress human rather than animal 
physiology. More importantly, the new ones are much less "directional, " 
As students build the models, they are not simply told what to do, but are ■ 
led through a series of "thought" questions about what they are doing and 
about the workings of the system they are modeling. In addition, once the 
models are built there are subsequent activities in which they are used, 
e, g, , teaching another student a concept using the model. These subse- 
quent activities were not part of the first set of Min-Exe, 

Klopfer has said that the IS project's evaluative criteria in such cases 
are threet Hie materials must "appeal to the kids, " match our goals, and 
meet an "indefinable kind of test, " namely, relationship of any part to the 
total program, 

Iliis holistic approach is also visible in the internal project organiza- 
tion of IS, Thm project is led by two co-dlreetor s. Observation and staff 
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interviewi reveal a consigtent form of leaderihip- -a division of taaki by 
time rather than taEk attribute; that is, both project leaderi are involved 
in all phaees of IS activity but one on some units, one on another. They do 
not duplicate each other. Decision making is perceivcdp and appeare to an 
obeerver, to be totally mutual and by agreement between the two. Staff 
cohesion ie a matter of some emphasis, with the project dirQctors attempt^ 
ing to convey a particular style or model of activityp 

Problem-Detection Procesyes 

As we have seen in examples from both the Quetelet and Harvey units, 
the processes by which problems are detected are very similar to those 
described in the Champagne and Klopfer paper (1974). They are somewhat 
''stagelike. Initial problem detection is accomplished during conceptualiza^ 
tion and early writing ('^Stage A''), At that stage, detection appears to be 
based on some inner mechanism of the project co-directors, derived from 
their cumulative experience* As the students work with prototype materials 
(''Stage B"). they are observed, talked with, and their written work is ana- 
lyzed- There ie a marked absence of reliance on test scores to reveal mucl 
about the program's effects up to this point. Instead, the developers seem 
to rely upon gathering information from as many of the involved actors as 
possible. 

The session at LRDC was itself a search for evidence that might dis- 
■confirm the initial successful set of experiences with the unit, Tliere was 
a high degree of eonsensus in the meeting on Quetelet, regarding both what 
^ was "good" and what required some revision, Tlie source of ttiis consensus 
■ may be found in the commonality of values of those involved. Though perhai 
-varying in degree by role, all those at the session shared a value set which 
emphasises tiie quality of a student's experience in a learning situation. 
This concern for people, for the human as well as the technological aspects 
of science, is also exemplified in the curriculum. The choice of a scientist 
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name to title each unit is a significant one, and a contraet to the norm, 
IS's units are entitled Quetelet and Harvey, not Statistics and Circulation* 

In Sister Martha, IS has a teacher whose values are congruent with 
those of the developers, so that consensus was achieved by agreement 
rather than by authority^ However^ it must be said that IS is functioning 
well (as defined by acceptance and maintenance of the program) in a large 
number of schools with a diversity of teachers. Even in the prototype 
stage^ IS has been handled by teachers with several different value systems 
As Champagne noted based on her observations in several prototype settings 
IS appears to have enough flexibility to suit the needs of a variety of teacher 
Consistent with the IS philosophy, if a teacher wished to strongly emphasize 
scientific literacy he/she would have the material for it in IS and could 
create that kind of environment* 

Problem -^Solving Processes 

Problem solving in curriculum, design is a creative process. It is a 
recasting, reformulating, or recreating activity which when completed 
results in modification of program content or teaching strategies. In the 
initial stage* problem solving is effected through editing, A great deal of 
this form of problem solving takes place before &e prototype materials 
are completed. 

Problem solving following prototype testing involves several strate- 
gies^ Once a trouble spot has been found, it may be decided toi (a) make 
no alteration, as the rationale for inclusion-as-is still appears sound; 
(b)^make some alterations, anything from minor editing to a major change 
(e*g, , the total rewrites of the Min-Exs in Harvey); or (c) withhold decision 
pending more student experience with the material. 

In the prototype testing of Quetelet, each of tiiese strategies was used 
in relation to some particular problem. It was decided, as in the case of 
the nonuse of some manipulables, to retain some instructional elementi 
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which could conceivably have been droppedi It was decided to make some ' 
changes in the managemant of Que tele t, auch as ipicLfying the timing of 
student- teacher conferences* In other places, vocabulary waa changed or 
its context modified to enhance clarity. Finally, some questions (es g* * 
the one concerning students' appreciation of the relevance of the content) 
were left open pending *Tiore student work Li Que toilet* 

Manifest and Latent Functioni of Forinatfvs Evaluation 

Hiii entire process of formatlvfij evaluation and most especially the 
- focal 'part of it, the meeting at LRDC, sevves both manifest and latent func- 
tions for those involved. At the manifest, or overt level* there is first the 
stated purpose* Formative evaluation helpe in the redesign of curriculum 
materials and procedures before these are field tested* Many changes have 
been made in past units of IS baaed on formative evaluation results; even in 
the unproblemmatic Quetelet unit, some changes were recommendedp Addi- 
tionally, where there is a question* and the decision is made not to alter 
the material, ttiie too serves a function* Should the problem reappear In 
field testing, and the evidence for change still not be considered persuasive 
,by the co-developerSj then the forethought and rationale for inclusion worked 
out in formative evaluation might well stand them in good stead in their deal- 
ings with others in outside agencies* 

This, in fact, is a second manifest function of the process. That is, 
the IS project must deal with two outside agencies before materials are 
finally completed I RBS* the field test agency; and Imperial Inter national 

V Liearning Corporation, the publisher, distributor, and marketer of ^e 
finished product* That a careful in-house review is conductedj Involving 
both the classroom teacher and evidence of students* work in the unit, 
strengthens the hand of LRDC staff in working with these two outside organi- 
zations. Both RBS and Imperial have on their staffs people with some back- 

• ground In science education who propose modifications in IS. In negotiating 



the form □£ the final product^ the LRDC etaif frequently need to mar shall 
DUpport for their position^ and formative evaluation resulti are useful in 
diis regard* 

In addition to these overt functions^ formative evaluation serves sev- 
eral latent functions* As a social process, it serves to validate and con- 
firm the models for action which the IS project co-directors have laid out. 
By "models for action" we mean both the overall instructional model of IS 
as well as the model for behavior within the project and in die project-s 
dealings with other actors and groups* l^ils model conceives of activity 
as holistic* In the analyslsi congruency of parts takes final primacy over 
excellence of pieces I all actors take several roles and become involved in 
several aspects of activity* 

It is possible to conceive of an alternative, "atomistic" model of 
behavior. In a project so structured, one might expect to find a highly 
differentiated division of labor i with an accompanying set of highly speci- 
fied roles arranged in a clear authority hierarchy* Interaction witiiln die 
project and between project and otiier groups would tend to be fairly formal 
and specific* Similarly, curriculum design and evaluation would place pri= 
mary importance on the quality of individual pieces developed radier than 
overall program configuration- 
nils is not ttie way the IS project functions* Hie project directors 
have a strong, shared model of appropriate action and it is their style of 
action which was modeled in the long evaluation session^ with its many 
participants. There appears to be emphasis on the IS staff adopting this 
model. For example, in a hypothetical case in which a staff member 
wished to rely heavily on analysis of pre- , and posttest scores, one could 
predict that this would be found unacceptable and that the rationale behind 
such an argument would be forcefully rejected. 

Hie formative evaluation process inspires self-confidence among the 
actors involved in the several roles that riiey play* Regardless of the 
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• ambtint of revision called for, the procese itself tendg to reinforce the posi- 
r/tive aspects of development, i. e, , ''w© ^have- a -good- ^ing -going -how^^ can- 
: we -together -make-it-better 7 " 

For the classroom teacher, a special purpose is served by this process 
- namely, the provision of a peer group. To be a science teacher in an ele- 
1 mentary school implies, in almost all cases, that the teacher is quite alone, 
at least in terms of having anyone with whom to share science education con- 
cerns. Especially in the case where the teacher demonstrates a willingness 
to be innovative- -as she/he does by taking on a program under development- - 
there is increased need for professional reinforcement. Meeting at LRDC 
: and being asked for advice and guidance increases the teachers' professional 
A- self-concept. 

Finally, fulfilling anoflier latent function related to a holistic approach 
to activity, the session at the Center brings together as many of those who 
have worked on the unit as is possible. In the case of Quetelet- -and probably 
others-- several of these people were soon to leave the role situations they 
had inhabited. ThiB "one last rallying" around a common task builds project 
cohesion for ttiose who remain and provides a sense of satisfactory closure 
for those departing. At the social Interaction level, such celebratory occa- 
sions are quite important. 

TTie Formative Evalu_atlQn Processi Functions and Structures 

In spite of the insights gained from our sttidy of the formative evalua- 
tion of the Quetelet unit, It must be said that overall our material Is "thim 
The unit was too good, the process too problem-free to be an ideal case for 
investigation. It has, however, led us to consider the formative evaluation 
process, its aspects and possible structures, beyond the specific case of 
Quetelet* 

Formative evaluation involves three process-like aspects; (a) techni- 
cal critique^ (b) problem posing, problem solution, and program improve- 
ment; and (c) social validation for involved actors. The first two of these 



are tnanllestp the third involves latent functions of the process. All aire© 
are complementary requirements of the process and a delicate balance 
among them is eaaentiaL 

Hie first two of these are temporally and logically sequenced, Actori 
must move from a critical stance vis-a-vis the object of evaluation to a 
creativep positive stance. Given this sequence, the form of the technical 
critique and the way in which its results are conveyed affects significantly 
the form of the problem aolutionp the creative aspect, l^m conduct of both 
of these, in turnp affects the validity of the process as a mechanism for 
social validationp though this function is not temporally ordered but is 
simultaneous wi^ the other two. 

It would be possible to structure a formative evaluation process in 
such a way that the technical critique overwhelmed the social validation 
possibilities and severely constrained tiie creative aspects of redesign, 
Similarlyp the creative aspect of the process could so be emphasized as to 
obviate the effects of technical criticism. This would be the case in which 
something was so aesthetically pleasing to the designer tiiat It was retained 
in ttie curriculum in the face of obvious negative evidence regarding its 
Inclusion. Finally, the process could be overcommitted to validation and 
confirmation and could turn into a mere celebrationp ignoring critiqua and 
redesign. 

Acknowledgini these potential dangers in the process, the question 
arises as to the appropriate structure for a formative evaluation. We do 
not wish to imply that one, and only one, structure is adequate. However, 
whatever structure is adopted must take the concerns we have outlined into 
account. One possibility that occurred to us is that, given its close connec- 
tions with other creative aspects of design work, formative evaluation should 
be congruent with these, lliat is, in the case we have looked at, where a 
holistic approach is taken to both curriculum design and project interaction, 
an atomistic approach to formative evaluation would be undesirable and 
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inappropriate were it indeed possible^ Thii kind of problem is not likely 
to arise when developers eonduet formative evaluation* But where tfisse 
two functions are fulfilled by different people, it becomes advisable for 
the evaluators to be sensitive to the designer's ''style" in such matters* 

In any formative evaluation, structure will vary wifli the answers to 
these questions! 

1* Who controls the process = = developers or evaluators? 
2*. Which organizations are involved in which functions? 

3. In what manner is the data from technical critique fed into 
the creative redesign proceBs? 

Whatever social design is chosen for formative evaluatlonj it must be one 
which enhances creativity while making criticism clear* iTiera is a close 
connection between creativity and validation; and a serious problem^ we 
suspect, is to allow critique to overwhelm the other two* 

Another aspect of formative evaluation is the necessary articulation 
of at least several frames of reference- -at minimum* those of the designer 
iJia evaluator* the classroom teacher , and the students in the classroom. 
Individuals* frames of reference are their means of orienting themselves 
to the realities fliey encounter, of constructing the meanings of events* 
Among this variety of actors, each possesses a frame of reference, some 
expert and discipline-based, others less reflective* But for each there is 
a set of values, priorities, and confirming tests, so diat variation among 
them IS inevitable* How they are translated one into the other and what 
their c'dmmon meeting grounds are to be must therefore be a concern in 
designing and conducting a formative evaluation* 

In the formative evaluation of Quetelet, the question of frame of refer 
ence was obviated somewhat by the fact that In the prototype phase, the IS 
co-directors conducted the process under their own aegis and within their 
own model of activity* Additionally * Sister Martha provided Quetelet with 
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a classroom teacher whose view differed little from fee IS modeL We feel 
that, as complex and rich a program as is IS, the formative evaluation pro- 
cedures that have been developed are well suited to the cognitive styles of 
the dominant actors and the curriculum itself. 

We close with a recommendation. There is a strong argument to be 
made for the inclusion of drastic variations in perspective at the formative 
rather than summative stage. That is. it might be of great interest, and 
highly prsductive for problem detection, to attempt even prototype forma^ 
tive evaluation in a setting or with perBOnnel seemingly quite unsuited to 
the IS model. Since this program is designed to be disseminated nationallyi 
and it has in fact been introduced to over 150,000 children already, clearly 
there will be implementation variations among classrooms. Subjecting the 
program at the prototype evaluation stage to severe "outside" critique might 
highlight problems which remain hidden in environments such as the one in 
which the Quetelet program was initially evaluated. 

We think that diis kind of risk taking can only be undertaken when a 
store of confidence in the product and its style of design already exists* 
Such confidence clearly exists regarding IS, and in later formative evalua- 
tions* or in iterations of the curriculum for other purposes or settings, 
such risk taking ought to be considered* 
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